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ATM INVERSE MULTIPLEXING 

Technical FIpIH a nd Indushnal App licarinn 

The invention relates to a new mechanism for sending ATM 
5 cells transparentiy over multiple slower transmission links In 
particular, the invention is directed to a method of inverse 
multiplexing of a series of ATM cells transparently over N 
transmission links (N being a positive integer) of slower speed. 

10 Back^ound Art 

It has been recognized tiiat the Tl/El rate (1.544/2.048 Mbit/s) is a 
cost effective way of user access to an ATM network as well as 
connection between ATM network switches. However, as ATM 
technology for wide area networks is deployed more and more 
demands for transmission links of a rate higher tiian Tl/El are 
increasing. Links of higher rates, such as T3/E3 (44.736/34.368 Mbit/s) 
have been designed to meet these needs. However, the cost of T3/E3 ' 
links is still prohibitive in many cases and the ratio of cost versus 
realistic utiUzation of the entire rate is not always attractive and fuUy 
justified for new ATM end users and service providers. ATM inverse 
multiplexers (AIMs) have been proposed to satisfy the need by using 
multiple Tl/El links which are grouped collectively to provide the 
service at a higher rate. 

Figure 1 and Figure 2 show two sample configurations in which 
AIMS are used. Figure 1 depicts a user access to a network through 
user network interfaces (UNIs) and Figure 2 a link connection between 
ATM switches through broadband inter-carrier interfaces (BIOs) or 
private network to network interfaces (PNNIs). 

Referring to the figures, the basic function of AIMs is to work in 
pairs to take an ATM ceU stream coming from tite ATM layer, send it 
over the multiple links by spreading cells over die available links and 
ensure tiiat the initial ceU stieam can be retrieved at the far end. Thus 
the AIMS preferably make tiie ATM ti-affic transparent to tiie ATM 
layer over multiple links which connect Uiem. As far as the ATM layer 
IS concerned, it should only see a pipe whose rate is now the sum of the 
multiple link rates. It is assumed tiiat each link is run in clear-mode 
witiiout the presence of intermediate ATM nodes processing ATM 
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cells. This means that there should be no cell discard by any 
intermediate transmission equipment. 

Currently no ATM inverse multiplexing protocols have been 
proposed which can properly interwork existing ATM inverse 
5 multiplexers or other ATM products which are already available on the 
market^ and yet are flexible enough to fit into the current standard 
ATM sp>ecifications. Two proposals for an ATM inverse multiplexing 
protocol have so far l>een made and are described in detail below. 

10 New Transmission Convergence Protocol Using GFC Bits 

This protocol was presented in 'Thysical Layer Sub-Working 
Group ATM Fonim/94-0775, ATM Inverse Multiplexing Mechanism", 
September 1994, by StrataCom Inc. The protocol robs two of the 
Generic Flow Control (GFC) bits contained in each cell transmitted 

15 over the multiple Tl/El links to implement a new transmission 

convergence (TC) layer. Figure 3 shows the ATM cell structure which 
is defined in the ITU Recommendation. The TC layer is defined by one 
GFC bit for framing and the other one for link control. The framing bit 
is used to determine relative link delays while the link control bit is 

20 used for commimication, control and administration between two TC 
points at two ends of the inverse multiplexer. 

In order to establish the sequence of cells over the links in a 
round robin manner, one end is defined as being "master" and the 
other as "slave". The "master" decides and informs the slave about the 

25 multiple link configuration using the control channel implemented 
through the link control bits. 

This protocol is oiUy applicable, however, for UNI application 
points because GFC bits that are robt>ed to implement the TC layer are 
only present in a cell defined for UNI. For NNI cells, the 

30 corresponding bits are no longer available since they are captured 

under the VPI field. Service providers are interested in ATM inverse 
multiplexers for carrying ATM traffic at rates higher than Tl/El and 
lower than T3/E3, but this protocol will not satisfy their need. It 
should also be noted that the protocol calls for a need to identify a 

35 "master" and a "slave" TC point and that requires an additional setting 
to be performed by the network operator. 
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Bit Pipe Inverse Multiplexing 

This protocol was presented in 'Thysical Layer Sub-Working 
Group ATM Forum/94-0956, Inverse Multiplexing of ATM cells over 
low speed UNIs such as Tl and El", September 1994, by Digital Link 

5 Corporation- It proposes a "bit pip>e" inverse multiplexing technique 
requiring the definition of a "bonding" (bandwidth on demand) like 
specification for N (positive number) Tl/El inverse multiplexing. 

It is not clear in the proposal how both ends of the links 
exchange information concerning the order of cells to be transferred 

10 from one end to amother over multiple links. The proposal mentions 
the existence and deployment of physical layer protocols that perform 
inverse multiplexing. The inverse multiplexer which can be used in 
this prop>osal is presumably the one defined by Digital Link 
Corporation in their "DL3800 DSl Inverse Multiplexer Users Manual, 

15 1993". 

The inverse multiplexing protocol defined in the above user's 
manual relies on the definition of an extra bit taken from Tl /El 
payload bits to configure the multiple links and adjust differential link 
delays. This protocol introduces the need for extra processing of data 

20 between devices dealing with Tl/El frames and ATM cell delineation. 
It also causes the ATM cells to no longer be byte aligned with the 
DSl /El frame. This is a requirement by the ATM Forum UNI DSl /El 
Physical Layer specifications. Changes like this would not be welcome 
by end users, vendors and service providers who are already using and 

25 deploying ATM equipment. 



Objects of the Invention 

It is therefore an object of the invention to provide a method of 
sending ATM traffic over a connection consisting of a plurality of 
30 transmission links. 

It is another object of the invention to provide a method of 
sending a series of ATM cells spread over a plurality of transmission 
liiUcs in a specific rovmd robin order. 

It is yet another object of the invention to provide a method of 
35 sending ATM cells containing sequence niunbers therein over a 
plurality of transmission links in a specific round robin order. 
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It is a further object of the invention to provide a method of 
sending a series of ATM cells between ATM inverse multiplexers over 
a connection consisting of a pluraUty of transmission links, transparent 
to the ATM layer. 

5 It is yet another object of the invention to provide a method of 

sending ATM traffic over a connection which consists of a plurality of 
transmission Unks and has been reconfigured after the connection 
start-up. 

It is still another object of the invention to provide an ATM 
10 inverse multiplexing method which is applicable to UNIs, Bias and 
PNNs. 

It is a further object of the invention to provide a method of 
preserving hnk integrity by periodically sending sequence number 
cells. 

It is a further object of the invention to provide a method of 
handling link failure and link reconfiguration. 

Disclosure of the Invenrion 

Briefly stated, according to one aspect the invention is directed to 

20 a method of sending digital data from a source node to a destination 
node over a connection consisting of a plurality of transmission links, 
the digital data containing a series of ATM data cells. The method 
comprises steps of the source node informing the destination node, at 
the connection start-up by way of ATM SN cells, of a specific round 

25 robin order in which the series of ATM data cells are to be transmitted 
over the connection, and sending each ATM data cell in the series to 
the destination node in the round robin order. 

According to a further aspect of the invention, the method 
comprises a further step of the source node sending ATM SN cells 

30 whose sequence number field is set to indicate the specific round robin 
order. 

According to yet another aspect, the invention comprises 
repeating the steps which have been performed at the connection start- 
up, wherever the connection is to be reconfigured. 
35 According to still another aspect, the invention comprises a step 

of a node informing another node of a link failure and repeating the 
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* Ihh TJT"^^' ""'"f"* » the even, that a link h« to be 

. Z b 1 "°' ""'^ f NNU and Bids. 

It must be transparent to the devices han,ii.«« ..u 

5 ATM ceUs into LpDHsignaI^ ^**'~"*'^"'*°' 
. It must be transparent to the devices dealing with AIM layer cells. 

itie present invention achieves all nf fho aK««^ 

, ^ «-»ucvcs ail or tne atx>ve requirements 

and »Wes fte problems discussed earUer. The invendon'reUtes to a 
10 lale 7""l ■""'HP'exing scheme that makes use of a physical 
10 layer opera^on administration and maintenance (OAM) cell wLh has 

mformabon to aUow proper operation of the ATM inverse 

The newly defined OAM ceU is caUed an AIM OAM ceU or AIM 
Sequence Number (SN) cell and is mainly designed to carry a cell 

sequence number and a feedback link status field. The sequence 

number in the SN ceU is made available for the receiver end for 

recovering .he initial cell stream from the incoming links. The 

feedback Imk status is made available for the receiver to inform the 

transmitter, by sending its own SN cells, that i, is receiving ceUs and it 

?K ".^ °' """" "-chanlsm, that is to 

ay. th^ feedback link status value SN «Us in either directly must 
agree wuh each other, although the sequence numbers a, both ends 
may be different. When the receiver sends its own SN ceUs in 
response to the transmitter that it is receiving cells, it is in fact 
acloiow edging that the receiver is ready to receive subsequent ATM 
ctatci cells. 

Figure 4 shows how the ATM ceUs are multiplexed and then 

transmitting node, 
an AIM 10 takes a series of ATM ceUs from an ATM layer devi^ It 
spreads ATM cells and transmits each ceU over each of N transmission 

r T""""' '^^ transmission is in round 

robm fashion. This process is called an inverse multiplexing. At the 
receivmg node, cells from N links are inverse demultiplexed 
(assembled) and sent to an ATM layer device by an AIM 14. The same 
order must be employed at this node to recover a proper sequence of 
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cells. Both nodes must be aware of the round robin order which is to 
be employed. Upon initialization, therefore, both AIMs send a series of 
SN cells in round robin fashion over the links, e.g. Tl/El links. This 
allows the receiver AIM at both nodes to establish the sequence in 
5 which to read cells from the incoming links as well as to adjust relative 
link delay. 

As mentioned above, the SN cell also carries an extra field used 
by each Tl/El link to indicate that both AIMs belong to the same link 
round robin. This information can then be used locally to determine if 

10 at the corresponding local node a link should be added, removed or 
maintained in the round robin. 

When a change of link configuration occurs by a link being 
added, removed or declared as being down, each node sends a series of 
SN cells to allow the far-end node to reestablish the sequence of cells to 

15 read from the incoming links. 

The protocol of the invention calls for a physical layer QAM cell 
that is defined to be exclusively processed by the ATM inverse 
multiplexers. The new cell structxire has to be consistent with the cell 
structure defined in ITU Recommendation 1.361. Figure 3 shows that 

20 structure of UNI/NNI ATM cells that is defined in 1.361. ITU 

Recommendations 1.361 and 1.432 state that ATM cells consisting of a 
header whose VPI and VCI fields are set to zero are reserved for use by 
the physical layer. So far three preassigned values of the cell header are 
reserved for use by the physical layer. They are shown in Figxire 5. 

25 The physical layer OAM cell according to one embodiment of the 

invention is then defined by using a non-assigned value by setting the 
PT field to "111". This is shown in Figure 6. 

The AIM OAM cell payload is then available for exchemging 
information between AIMs. The cell payload consists of a series of 

30 fields whose locations are shown in Figure 7. 

The following fields are identified for the AIM protocol 
according to one embodiment of the invention: 

• ATM inverse multiplexing remote defect indicator (AIMRDI) - one 
octet is allocated and the propoised coding is all "1". 
35 • ATM inverse multiplexing far-end receiver ready (AIMFERR) - one 
octet is allocated and the proposed coding is all "1". 
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• ATM mverse multiplexing ceU sequence number (AIMCSN) ' 

the same as fhaf rtf fK« n i- w wxiuiuiu , which is 

~iiicr as mat ot the idle cell as DroDo<;pH mr o 
L361. proposed in ITU Recommendation 

15 only shows one direcUon for dari^l. ^.T deUU b^low. The figure 
links are all Tl/El links b^^^^ . ^ *" """-ni^ion 

Ti,H,. as^tin^-rjs- 1^ s^r 

^0 the .ansnu«ng nodeTp^TaXov^r J^.;:'Zr:!TT '"^^ " 
to a receiving node 26. An AIM at the r^^"^! --ansmission 
ATM data eeUs received from th. i T """^ ««s=mbles 

to the ATM layer ^ " ""P" ^-d' *em 

25 Link StarKi.p 

.vailahi^^rS ""^^^^ 

on the order in which lrh, . K' ^""'<="'8 cells is based 

30 (composed o N Xsi^tlT l ""T"*" *^ 

recervmg node queues the received AIM SN cells ™m?r^^' 
the sequence in which to read the ATM data ^ u T '! 

3 -^asweU as the diHerential delay™ »gi^:,:::;:i:~% 
one for each link which is now considered "ready to rec^ve I™ 
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traffic. From that moment, the receiving node knows the sequence of 
cells coming from the links. A link is being considered available if cells 
are currentiy delineated and AIM-RDI is not received on the incoming 
link. 

5 When the transmitting node starts sending the AIM SN cells, it 

starts a time-out of 100 milliseconds within which the receiving node 
determines the sequence of AIM SN cells which it is receiving from the 
links. The time-out expires linless the transmitting node receives the 
"ready" signal from all the links which were considered available at the 

10 initialization. 

If there is no "ready" link when the time-out expires, the local 
node reevaluates the availability of the links (using cell delineation), 
starts sending AIM SN cells over the available links and re-starts the 
time-out. This procedure is repeated imtil at least one avciilable link is 

15 declared ready. 

When the time-out has been canceled due to the reception of the 
"ready" signal from all the available links, or when there is at least one 
link ready when the time-out expires, the local end starts sending ATM 
layer cells over the "ready" link or links using the same round robin 

20 order used at the time of initialization. 

Each ATM layer cell gets a sequence nvimber assigned to it, but 
only the AIM SN cells carry that number across the links. After both 
nodes have started sending ATM layer cells, they periodically send a 
series of "n" AIM SN cells over links to allow the receiving node to 

25 readjust the differential delays among the links. The value of "n" is 
equal to the number of "ready" links used to carry ATM layer ceUs. 
This means that the receiving node always scans for AIM SN cells in 
order to determine if it is always reading cells from the multiple links 
in the correct order. How often a burst of "n" AIM SN cells is sent 
30 depends upon the link utilization but the maximum period has been 
set to 50 milliseconds in one embodiment. An example of cells sent 
over a virtual link composed of a plurality of transmission links is 
shown in Figure 9. 
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Link Reconfiguration 

As mentioned above, ti\e protocol according to the invention 
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also deals with link reconBguration. There are three possible cases in 
which reconfiguration can occur: 

• a new link has to be added to the round robin; 

• a link has to be intentionally removed from the round robin; and 
5 •a link has been declared inadequate to provide service (e.g. link 

down by failure etc.). 

In the first two cases, a node starts link reconfiguration by 
performing the same initialization process for a connection start-up, 
that is to say, it chooses a round robin order among the links and starts 

10 sending AIM SN cells using the chosen rotmd robin order. The 
receiving node, seeing the occurrence of AIM SN cells, will stop 
transmitting traffic and starts sending AIM SN cells of its own while 
adjusting itself to receive traffic from the incoming links. The rest of 
the protocol is as described earlier for start-up. 

15 Two possible cases for reporting failure conditions are: 

1) A medium problem reported through Loss of Delineation 
(LCD) failure condition. In this case, the corresponding link should not 
be used for service. When detecting LCD, SN cells with AIMRDI set to 
"1" should be sent over the corresponding outgoing link. SN cells 

20 containing AIM-RDI do not carry a valid sequence number. At the far- 
end, the detection of AIM-RDI signals vriU indicate that the link is not 
to be xised. Therefore, if LCD or AIM-RDI is detected, the link is no 
longer considered available and won't be allowed to be part of the 
round robin on both sides. When this has happens, SN ceils are re- 

25 sent over the remaining links to re-establish the connection between 
the two ends. 

2) Cells are lost without an LCD or AIM-RDI being reported. For 
instance, this would occur when a few cells are discarded by the 
physcial layer device due to bit errors in cell headers. This wotild cause 

30 the cell sequencing to be affected. One symptom would be the detection 
of SN cells whose number is no longer the same as that expected (since 
one or more of the previous cells are missing). In this case, the 
problem might be partially corrected by re-ajusting the receiver 
buffering system (assuming the difference between the SN cell number 

35 and the expected number is small). The other symptom would be the 
absence of a SN cell on one link when getting a burst of SN cells on all 
the other Unks. In that case, the local end would have to force the links 
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to be re-configured. If the symptoixts described above reoccur over a 
given period/ the bad link may have to be removed from the round 
robin. 

A link reconfiguration also occurs when it tak^ too much time 
5 for one link to receive cells from the far-eiid (receiving) node^ that is to 
say, no cells within^ e.g., 32 milliseconds. 

Cell Sequence Number Rang e 

Because an ATM inverse multiplexer must absorb a differential 
10 delay between individual links of up to 32 milliseconds in one 

embodiment, and because the system needs to deal with a maximum of 
8 Tl/El links, it is necessary to have a sequence number whose modulo 
is large enough to accommodate such delay. 

As a practical example, the following parameters for El are 
15 considered: 

• full rates: 2.048 Mbit/s 

• payload rates: 30/32*2.048 Mbit/s = 1.92 Mbps 

• ATM cell: 53 bytes 

• ATM cell period time: 221 microseconds 
20 • ATM cells /32 milliseconds = 144 cells. 

For a delay of up to 32 nulliseconds on each link, there is a need 
to queue cells for at least the same length of time on each link. 
Therefore, in this example, this means that a delay of up to a period of 
144 cells between two links is possible. An ATM inverse multiplexer 

25 can handle a maximum of 8 Tl/El links. Therefore up to 1008 cells 
(144*7 cells) must be queued at one time by one ATM inverse 
multiplexer. This requires a sequence number modulo large enough to 
cover this scenario. A simple case in one embodiment is to use a 16-bit 
count that has modulo 65536. 

30 The ATM inverse multiplexing protocol according to the 

ihviention realizes the following characteristics: 

• it is applicable to UNIs, BICIs and PNNIs (any applicable points in 
an ATM network); 

• it does not affect the ATM cell header of currently defined cells; 
35 • it does not require a change to current ATM physical layer devices 

dealing with the convergence of ATM cells into Tl/El; 

• it operates transparently to the ATM layer; 
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it does not require a change to current devices dealing with the 
processing of the ATM layer cell; 

it is self-configuring among available links upon start-up and self- 
reconfiguring among the multiple Unks in the case when a new 
link has to be added, deleted or considered inadequate to provide 
service; and 

it includes a sequence number whose modulo is large enough to 
meet the requirement of a large differential delay among the links. 




# 



wo 96/17489 



PCT/CA95AM»22 



5 



i 



10 



15 



20 



25 



WHAT IS CLAIMED IS: 

1. In a method of sending digital data from a source node to a 
destination node over a connection consisting of a plurality of 
transmission links, said data containing a series of ATM data cells, the 
invention being characterized in steps of: 

at the connection start-up, the source node sending to the 
destination node ATM SN cells informing the latter of a specific order 
in which the series of ATM data cells are to be transmitted over the 
connection; 

the destination node sending ATM SN cells whose receive ready 
field is set to indicate that the destination node is ready to receive ATM 
data cells in specific round robin order from the plurality of 
transmission links; and 

the source node sending each ATM data cell in said series to the 
destination node in said round robin order. 

2. The method according to claim 1 wherein the step of the 
source node informing the destination node includes a step of the 
source node sending a series of ATM SN cells whose sequence nxmtber 
field is set in a sequential order, each ATM SN cell being sent over each 
transmission link in a specific round robin order. 

3. The method according to daim 2 wherein the step of the 
destination node sending the source node includes a step of the 
destination node sending a series of ATM SN cells whose receive ready 
field is set and whose sequence number field is set in a sequential 
order, each ATM SN cell being sent over each transmission link in a 
specific round robin order. 

4. The method according to daim 3 comprising a further step of 
the source node and destination node sending periodically to one 
another a series of ATM SN cells over the transmission links in 
spedfic round robin order within a predefined time period. 
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5. The method according to claim 4 wherein said ATM SN cells 
are physical layer OAM cells whose protocol fields are properly set to 
indicate types of cells and said sequential order. 

5 6. The method according to claim 5 wherein said ATM SN cells 

are physical layer OAM cells whose protocol fields contain a field to 
indicate the destination node is ready to receive ATM data cells. 

7. The method according to claim 4 wherein the source node 
10 and destination node send periodically a burst of "n" SN cells to each 

other over "n" transmission links, within a predetermined maximum 
time period. 

8. The method according to claim 7 wherein the period of the 
15 source node and destination node sending a burst of "n" SN cells to 

each other over "n" transnussion links is adjustable within the 
maximum time period, depending upon the link utilization. 

9. In a method of sending digital data from a node to another 
20 node over a connection consisting of a pluraUty of ti-ansmission links, 

said data containing a series of ATM data cells, the invention being 
characterized in steps of: 

whenever the connection is to be reconfigured, a first node 
sending to tiie other node ATM SN cells whose sequence number field 
25 is set to indicate a specific round robin order in which the series of 
ATM data cells are to be transmitted over the connection; 

the other node, in response to the first node, sending ATM SN 
cells whose receive ready field is set to indicate the other node is ready 
to receive ATM data cells in specific round robin order from the 
30 plurality of transmission links; and 

the first node sending each ATM data cell in said series to the 
other node in said round robin order. 

10. A method according to claim 9 comprising further steps of: 
35 whenever the coimection is to be reconfigured upon failure of a 

trax\smission link, a node sending to the other node an indication that 
a link has failed; and 
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the two nodes performing the steps redted in claim 9 over the 
remaining transmission links. 

11. The method according to claim 9 wherein said ATM SN cells 
5 are physical layer OAM cells whose protocol fields are properly set to 

indicatie types of cells and said sequential order. 

12. The method according to claim 10 wherein said ATM SN 
cells are physical layer OAM cells whose protocol fields are properly set 

10 to indicate the link failure. 
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